Problem

LLMs are stateless, so our agents are dumb when questions require details. This PR gives them the ability to remember simple facts about a user's company or product in the core memory. The core memory is a text that the background agent continuously updates using two actions: appending a new fragment and replacing an existing fragment. The memory agent triggers on each new human message (surprisingly cost-effective).

Additionally, the PR includes another agent scraping the initial memory from the Internet. It searches in taxonomy for existing $host (preferable) or $app_namespace (app bundle id) and scrapes relevant sources using Perplexity to automatically fill the agent's initial core memory. However, this information is not always reliable, so we always confirm that we've scraped the memory correctly.

Demos

Changes

• Add onboarding for the agent.
• Scrape Perplexity for relevant memory about the user's company, product, business model, target audience, and features.
• Implement simple dynamic forms for confirmation/rejection of scraped info.
• Compress the scraped memory to a smaller text since Perplexity's output is very verbose.
• Implement the memory collector agent, which works as a parallel node on every new human message.
• Replace the product description with the core memory in the agent nodes.

Does this work well for both Cloud and self-hosted?

No

How did you test this code?

Manual testing, unit tests.